{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "mNESTCLfvSrY"
      },
      "source": [
        "# Welcom to SSCMA for Google Colab Training Example 🔥 \n",
        "\n",
        "<a href=\"https://colab.research.google.com/github/Seeed-Studio/SSCMA/blob/main/notebooks/Google-Colab-PFLD-Grove-Example.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a> **[🚀🚀🚀 One-Click to Deploy to Google Colab 🚀🚀🚀](https://colab.research.google.com/github/Seeed-Studio/SSCMA/blob/main/notebooks/Google-Colab-PFLD-Grove-Example.ipynb)**\n",
        "\n",
        "In this tutorial, we will demonstrate how to use [SSCMA](https://github.com/Seeed-Studio/SSCMA) and Colab to train a PFLD model for meter reading ⏱️.\n",
        "\n",
        "Table of contents:\n",
        "\n",
        "- [Setup SSCMA](#setup-SSCMA)\n",
        "- [Download Custom Meter Datasets](#download-custom-meter-datasets)\n",
        "- [Train PFLD Model](#train-pfld-model)\n",
        "- [Optimize PFLD Model and Export](#optimize-pfld-model-and-export)\n",
        "- [Deploy to Grove Vision AI](#deploy-to-grove-vision-ai)\n",
        "- [Thanks for Trying out SSCMA 🎉](#thanks-for-trying-out-SSCMA-🎉)\n",
        "\n",
        "For more details of PFLD model or application scenarios, please refer to [SSCMA Documentation](https://seeed-studio.github.io/SSCMA/tutorials/training/pfld).\n",
        "\n",
        "**Tips: Since model training consumes a lot of computational resources, we recommend that you run this notebook on [Colab's GPU instances](https://research.google.com/colaboratory/faq.html#gpu-availability).**"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "PRGi8-2jvxzR"
      },
      "source": [
        "## Setup SSCMA"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Ikzx6C6wrpos"
      },
      "source": [
        "**Step 0:** Clone the SSCMA source code from [SSCMA GitHub repository](https://github.com/Seeed-Studio/SSCMA), and enter the SSCMA project directory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F5pMIwXuvPpG",
        "outputId": "d66ee6b7-6398-4d0a-a78b-4cf294a4b4e2"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/Seeed-Studio/SSCMA.git  # currently we're using experimental 2.0 version branch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-4H83YK8waNA",
        "outputId": "1f8ea61f-132e-4d42-c336-d8c3eabbda8b"
      },
      "outputs": [],
      "source": [
        "%cd /content/SSCMA"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "LWAE--J9sAmW"
      },
      "source": [
        "**Step 1:** Use `python3.8` as default python in colab, then install corresponding version python and pip package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "STOKRHwXs70f",
        "outputId": "1afa4628-764c-4edf-bb13-261f3b0e4786"
      },
      "outputs": [],
      "source": [
        "%env PYTHON_EXEC=python3.8"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xF4nReUbqbUA",
        "outputId": "2bf0b960-152c-49ad-a4c8-79269a9c2a82"
      },
      "outputs": [],
      "source": [
        "!sudo apt-get update\n",
        "!sudo apt-get install ${PYTHON_EXEC}-dev python3-pip -y"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "zX3aW3Rys1Zr"
      },
      "source": [
        "**Step 2:** Use `scripts/setup_colab.sh` to automatically setup SSCMA dependencies."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k1HIXVgCzrCz",
        "outputId": "074841fc-12c1-4ead-a391-1a5bc2fe902e"
      },
      "outputs": [],
      "source": [
        "!chmod +x scripts/setup_colab.sh\n",
        "!scripts/setup_colab.sh $(which ${PYTHON_EXEC})"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "v47-pxMe2DP1"
      },
      "source": [
        "## Download Custom Meter Datasets"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "90NL7Kg-vxy7"
      },
      "source": [
        "First, download and unpack our custom meter datasets.\n",
        "\n",
        "For more available datasets, please refer to [SSCMA Documentation - Datasets](https://seeed-studio.github.io/SSCMA/tutorials/datasets)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oSJZdF4l2JEe",
        "outputId": "8bc4111f-ee71-4e08-ad0b-68c9a509d02b"
      },
      "outputs": [],
      "source": [
        "!wget https://files.seeedstudio.com/wiki/SSCMA/meter.zip -P datasets/\n",
        "!unzip datasets/meter.zip -d datasets/"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "2HMAOOWR2sf6"
      },
      "source": [
        "## Train PFLD Model"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "zmWPp53tcLpB"
      },
      "source": [
        "### Examine the PFLD config file\n",
        "\n",
        "The config we use for train is located at `configs/pfld/pfld_mv2n_112.py`.\n",
        "\n",
        "For more config details, please refer to [SSCMA Documentation - Config](https://seeed-studio.github.io/SSCMA/tutorials/config)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J7TrWv8_wjXY",
        "outputId": "1d6f4ec0-dec4-417c-908e-ed4314c73303"
      },
      "outputs": [],
      "source": [
        "!cat configs/pfld/pfld_mbv2n_112.py"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "l61rdIv_Z59j"
      },
      "source": [
        "### Download Pre-train weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lEN9rtggW7yt",
        "outputId": "688d94c9-a4a0-4265-809e-872cdb02772f"
      },
      "outputs": [],
      "source": [
        "!wget -P pre-train/ https://github.com/Seeed-Studio/SSCMA/releases/download/model_zoo/pfld_mbv2n_112.pth "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Y9sSOwrZdaLo"
      },
      "source": [
        "### Train the model using default config\n",
        "\n",
        "Then, train PFLD model using default config, note here we need to override the config by 2 parameters:\n",
        "\n",
        "- `data_root` - the datasets path, which located at path `datasets/meter`\n",
        "\n",
        "- `epochs`- the train epochs, we use `50` to reduce the training time\n",
        "\n",
        "- `load_from` the pre-train weights, make training faster\n",
        "\n",
        "For more training details, please refer to [SSCMA Documentation - Train PFLD Model](https://seeed-studio.github.io/SSCMA/tutorials/training/pfld)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zUgKXe2s2wto",
        "outputId": "48c35897-3936-4772-b9bf-c33b4efc04e0"
      },
      "outputs": [],
      "source": [
        "!${PYTHON_EXEC} tools/train.py \\\n",
        "  configs/pfld/pfld_mbv2n_112.py \\\n",
        "  --cfg-options \\\n",
        "    epochs=50 \\\n",
        "    num_classes=1 \\\n",
        "    data_root='datasets/meter' \\\n",
        "    load_from='pre-train/pfld_mbv2n_112.pth'"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "tGl9VBZavSmM"
      },
      "source": [
        "The output directory in the training process would be look like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ff8MtMaSu8Ij",
        "outputId": "06ce3142-4d4e-410b-9417-c87f1a4feed2"
      },
      "outputs": [],
      "source": [
        "!ls -alh work_dirs/pfld_mbv2n_112"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "m4Y6DM16vgLF"
      },
      "source": [
        "We could inspect the latest model weight path by this command:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RMf9sHP2vNzW",
        "outputId": "e0dceb1f-678a-4c3c-9d4c-86a7f324f5ea"
      },
      "outputs": [],
      "source": [
        "!echo \"Latest model weights path: $(cat work_dirs/pfld_mbv2n_112/last_checkpoint)\""
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "G7KqGi2OxnaM"
      },
      "source": [
        "### Validate the Model\n",
        "\n",
        "We recommend you to validate the model with the latest weight after training using the `tool/test.py` script."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v5SckZpKx21c",
        "outputId": "86e0d575-60ad-45de-fe3a-9a9efb86ffcd"
      },
      "outputs": [],
      "source": [
        "!${PYTHON_EXEC} tools/inference.py \\\n",
        "    configs/pfld/pfld_mbv2n_112.py \\\n",
        "    \"$(cat work_dirs/pfld_mbv2n_112/last_checkpoint)\" \\\n",
        "    --dump work_dirs/pfld_mbv2n_112/last_checkpoint.pkl \\\n",
        "    --cfg-options \\\n",
        "        data_root='datasets/meter'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5vk0o2HAyaz7",
        "outputId": "8341030e-3843-4e48-92ea-bc771d5bc746"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "\n",
        "with open('work_dirs/pfld_mbv2n_112/last_checkpoint.pkl', 'rb') as f:\n",
        "    data = pickle.load(f)\n",
        "    for k in data.keys():\n",
        "      print(f'{k}: {data[k]:.5f}')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "5np32KYq8JL4"
      },
      "source": [
        "## Optimize PFLD Model and Export"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "QF9ObhTrjwI2"
      },
      "source": [
        "### Export PFLD PyTorch model to TFLite model\n",
        "\n",
        "Here we're using `tools/torch2tflite.py` to convert and export the latest weights to a TFLite model at a INT8 precision for inference."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a50cBz7P8TP-",
        "outputId": "28ad830a-19f0-4707-91fb-25b7d74af05a"
      },
      "outputs": [],
      "source": [
        "!${PYTHON_EXEC} tools/export.py \\\n",
        "  configs/pfld/pfld_mbv2n_112.py \\\n",
        "  $(cat work_dirs/pfld_mbv2n_112/last_checkpoint) \\\n",
        "  tflite \\\n",
        "  --cfg-options \\\n",
        "    data_root='datasets/meter'"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "akUdGWE-1Z-8"
      },
      "source": [
        "## Deploy to Grove Vision AI\n",
        "\n",
        "This example is a tutorial for deploying the models from [SSCMA](https://github.com/Seeed-Studio/SSCMA/) to Grove - Vision AI module, based on the [Synopsys GUN Toolchain](https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain) and [Tensorflow Lite Micro](https://github.com/tensorflow/tflite-micro) implementations.\n",
        "\n",
        "For more details, please refer to [SSCMA Documentation - Example - Grove - Deploy](https://seeed-studio.github.io/SSCMA/examples/grove/deploy#compile-and-deploy)."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "1huhlfmt3np4"
      },
      "source": [
        "### Download and Setup Build Tool Chain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7hIBg6T73Ro3",
        "outputId": "2d364830-dbf8-4c3f-b02e-cc87689c0c68"
      },
      "outputs": [],
      "source": [
        "!wget https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases/download/arc-2020.09-release/arc_gnu_2020.09_prebuilt_elf32_le_linux_install.tar.gz -P ~/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HXrWQwVZ5I5O"
      },
      "outputs": [],
      "source": [
        "!tar -zxf ~/arc_gnu_2020.09_prebuilt_elf32_le_linux_install.tar.gz --directory ~/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WDBzyjFL7RHN",
        "outputId": "a48069c6-040d-4836-b5cc-a6f0f0681ed6"
      },
      "outputs": [],
      "source": [
        "SYSTEM_PATH = %env PATH\n",
        "SYSTEM_PATH = SYSTEM_PATH.replace('\"', '')\n",
        "SYSTEM_HOME = %env HOME\n",
        "SYSTEM_HOME = SYSTEM_HOME.replace('\"', '')\n",
        "SYSTEM_PATH = f'{SYSTEM_HOME}/arc_gnu_2020.09_prebuilt_elf32_le_linux_install/bin:{SYSTEM_PATH}'\n",
        "%set_env PATH={SYSTEM_PATH}"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ZMSi7Ie--O1h"
      },
      "source": [
        "### Clone and Setup Grove Vision AI SDK"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vD6cIBo9-WPA",
        "outputId": "aca85b23-795f-4e93-9a63-0187802c81cd"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/Seeed-Studio/sscma-example-vision-ai example/grove"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4J5Z4cwp-fne"
      },
      "outputs": [],
      "source": [
        "!cd example/grove"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SZ0UyC7HA1zH",
        "outputId": "a8d67db3-62b8-4ee6-9901-31211a5b6ab7"
      },
      "outputs": [],
      "source": [
        "!${PYTHON_EXEC} -m pip install numpy requests colorama serial pyserial"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "IGHvs_2n--gP"
      },
      "source": [
        "### Build Grove Vision AI Firmware Img"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "18TRCN-z-r3_",
        "outputId": "6483334d-e5c0-46fe-cbde-e16032bbed8f"
      },
      "outputs": [],
      "source": [
        "!cd example/grove && \\\n",
        "  make HW=grove_vision_ai APP=meter"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "6ISXUUG2_uKY"
      },
      "source": [
        "### Convert Firmware Image to UF2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OJ5lwvPk_zhh"
      },
      "outputs": [],
      "source": [
        "!cd example/grove && \\\n",
        "  ${PYTHON_EXEC} tools/ufconv/uf2conv.py \\\n",
        "    -t 0 \\\n",
        "    -c tools/image_gen_cstm/output/output.img \\\n",
        "    -o firmware.uf2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i3effVeWCAhS"
      },
      "outputs": [],
      "source": [
        "!echo \"The UF2 firmware is located at: $(pwd)/example/grove/firmware.uf2\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TFC2hSiXCfM3"
      },
      "outputs": [],
      "source": [
        "from IPython.display import FileLink\n",
        "\n",
        "FileLink('example/grove/firmware.uf2')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "2zrBJSg0FST3"
      },
      "source": [
        "### Convert Trained Model (TFLite) to UF2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P4-YcaURF6K9"
      },
      "outputs": [],
      "source": [
        "!cd example/grove && \\\n",
        "  ${PYTHON_EXEC} tools/ufconv/uf2conv.py \\\n",
        "    -t 1 \\\n",
        "    -c \"$(cat ../../work_dirs/pfld_mbv2n_112/last_checkpoint | sed -e 's/.pth/_int8.tflite/g')\" \\\n",
        "    -o model.uf2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lMqfOQteIEL5"
      },
      "outputs": [],
      "source": [
        "!echo \"The UF2 model is located at: $(pwd)/example/grove/model.uf2\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B-___RClH92g"
      },
      "outputs": [],
      "source": [
        "FileLink('example/grove/model.uf2')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "_4cVSi76EsFg"
      },
      "source": [
        "Please download the UF2 Firmware and UF2 Model and flash them into a Grove Vision AI module. Finally, you can see the real-time meter reading results as shown in the figure below.\n",
        "\n",
        "![Meter Reader](https://seeed-studio.github.io/SSCMA/static/grove/images/pfld_meter.gif)\n",
        "\n",
        "For more information on **how to use these UF2 images**, please refer to [SSCMA Documentations - Deploy - Grove - Deployment Routines](https://seeed-studio.github.io/SSCMA/examples/grove/deploy#deployment-routines)."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "qasiYT2qJWPm"
      },
      "source": [
        "## Thanks for Trying Out SSCMA 🎉\n",
        "\n",
        "Congratulations, you have completed this tutorial. If you are interested in more application scenarios or our projects, please feel free to give [SSCMA](https://github.com/Seeed-Studio/SSCMA) a star ✨ on GitHub.\n",
        "\n",
        "If you have any questions about this tutorial, please also feel free to [submit an issue](https://github.com/Seeed-Studio/SSCMA/issues)."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
